Bayesian exploration for approximate dynamic programming

Approximate dynamic programming (ADP) is a general methodological framework for multistage stochastic optimization problems in transportation, finance, energy, and other applications where scarce resources must be allocated optimally. We propose a new approach to the exploration/exploitation dilemma in ADP. First, we show how a Bayesian belief structure can be used to express uncertainty about the value function in ADP. Bayesian models can be integrated into both parametric and non-parametric value function approximations, which is vital for practical implementation. Second, we propose a new exploration strategy, based on the concept of value of information from the optimal learning literature, and prove that it systematically explores the state space. We evaluate this strategy using a variety of distinct resource allocation problems and demonstrate that it is highly competitive against other exploration strategies.